How is the hash's input created? Marhsaling the object itself? I've seen quite a few cases where a time or resourceVersion value is included in a hash/comparison, with an always-true result, so exposing such a function may be a source of confusion and bugs.

Firstly for each resource, you have to define in the "getHashXXX()" method, the scope of objects/elements covered by the hash computation. OKT provides you an helper to define this scope. Internally it use an array of pointers with all the elements participating to the scope (the metadata you want, the whole Object's Spec part if you want, actually whatever you want).
You return this "bag" of collected elements through the return statement of "getHashXXX()' method.

To perform the Hash computation, OKT use the spew library (github.com/davecgh/go-spew/spew) which follows pointers and prints actual values of the nested objects ensuring the hash does not change when a pointer changes.
The returned hash can be used for object comparisons.

So if the developper don't add resourceVersion in the scope, there's no risk to meet the bug you mentionned. I know that some immutable fields (IP in Services, ...) and the Secret's data are things to take care about (OKT helper help on this point).
There is an example in the operator Memcached with OKT I sent to you and Tony (it can be shared).

So it sounds like the hash is defined by the developer and they can choose as much or as little in their hash. And I agree with Eric that if someone unknowingly adds an attribute that changes it will cause an infinite reconcilation loop. (we've done this ourselves :)

if someone unknowingly adds an attribute that change it will cause an infinite reconcilation loop

OKT comes with an integrated throttling mechanism in case of a same recurrent errors, but not sure it can help in such case...
Besides that, when the developer defines the hash in the "getHashXXX()" method, he uses an OKT helper generic or dedicated for the resource kind he is dealing with. For example, the SecretMutationHelper object manages few things with StringData and Data fields. It ensures too that the hash computation covers only the Data field and not the StringData field.
Here again, I don't know if there's a more elegant solutions and if a such helper is sufficient to guarantee the problem of the changing attributes added unknowingly.
The counterpart of these helpers, implemented by OKT, is the need to maintain them. Today, the only specific one that exists (the others are generic helpers), is the SecretMutationHelper. I thought to need another one specific for the Service resource and its changing ClusterIP, but in practice all works correctly without special care, it seems it was a mistake to think that.

Edited the last sentence regarding specifics vs generics helpers.

Assuming you mean that the operator registers 3 types with OKT, this sounds a lot like controller-runtime's cache.

I must admit don't know lot of things about the controller-runtime's cache implementation.

At OKT level it is an array of a generic resource type, that OKT uses at the "Mutator" and "Updater" reconciliation steps to loop on each resource and call respectively the "getHashXXX(), MutateWithInitialData(), MutateWithCR()" (for mutation) and a specific "CreateOrUpdate()" OKT method.

I think the concept of a controller state machine makes a lot of sense, and one generic enough to register an arbitrary application lifecycle for an operator to execute would be powerful. I think such a library warrants an enhancement on its own, with a prototype ideally.

Yes thank you, the concept is interresting and I have some ideas to render the LC graph implementation generic at some points. Note that the K8S resources reconciliation steps (CRChecker(), ObjectsGetter(), ..., SuccessManager() ) are already implemented thanks to a StateMachine library (Qor transition), it is a bit overrated because initialy I thought about a more complex reconciliation process, but it's done with that right now.
It is clear now, that it is better to separate in 2 distincts processes the K8S resources reconciliation and the Application's LC graph stages.
Besides that, some ideas comes in mind. We could manage the state changes through a standard events formalizations as the one brought by the CloudEvents specs and lib. Providing a standard way to observe and pilot the state change seems interesting. So we already are looking for integrating CloudEvents in OKT for this purpose.

👍 agree that this state machine idea sounds quite interesting.

Can you clarify exactly what you mean by "depends on the OperatorSDK"? The operator-sdk repo contains a binary that scaffolds code from controller-runtime, with no public libraries; perhaps you mean OKT depends on the latter?

Yes there's no real dependancies except that OKT has no real sense without the usage of the OperatorSDK as OKT is done to be used inside an OperatorSDK's scaffolded project.
Besides that, each time the librairies used by the OperatorSDK evolves (sigs, ...), we have to align the OKT librairy as well.

Is this similar to the finalizer helpers present in controller-runtime - https://github.com/kubernetes-sigs/controller-runtime/tree/master/pkg/finalizer

Actually, the OKT Reconciler just check if a finalizer in CR Metadata with the same name as itself exists. If it's the case and the CR is being deleted, the stage "CRFinalizer" is called right after the stage "CRChecker" instead of continuing the normal process (ObjectsGetter, ...).
The principle is to do what you need in the CRFinalizer stage and call a provided method to delete the finalizer.
So it is just a way to manage a single finalizer on the CR.
I guess the helper you mentioned is generic as well to manage several finalizers...

Would it be useful to explain in detail on how the OKT Reconciler is different from the generic controller-runtime reconciler which is used in operator-sdk ? In the sense the aspects which are missing from the current implementation or the additional features which OKT Reconciler implements to fill the gaps.

Yes a good explanation by a sample would help. This the the reason I'll provide a more complete version of the Memcached operator sample. I believed Eric S. got a tarball with the first version of Memcached aligned with the "User Story 1" in this issue, feel free to go it.
Now, I'm very close to get the official GO to publish OKT on Github, I think there's no issue to drop a new Memcached operator sample under my Github space. I'll try to release it nex week.

This seems to be really useful. But just curious, would conditions be useful here to expose the status of CR. Please correct me if this is totally tangential to what is being described here.

I'm not sure to well capture your point but I guess you're wondering if the CR status is the right place to store the conditions that represent the application's state.
I had in mind the ability to offer a simple way to follow the application state thanks to a "kubectl wait" command for example. But actually it sounds a bit unsuitable to me too, so may be that was the reason of your question :).
So it's a point still unstated today. Besides that, building a specific interface relying on the CloudEvent api could be a better option either in term of interoperability with external world and may be for others internal needs too...